Model Selection in a Settingwith Latent Variables

Model selection, the task of selecting a statistical model from a certain model class given data, is an important problem in statistical learning. From another perspective model selection can also be viewed as learning a single distribution, where the parameter space includes a discrete structure parameter s which imposes further constraints on the remaining parameterization of that model, so that learning a distribution decomposes intro learning s and the corresponding probability parameters. Examples for model classes that offer this kind of structural flexibility and thus entail a model selection problem are Bayesian networks [1] (where s is the DAG), Markov chains (where s is the order), and variable order [2] and parsimonious [3] Markov models (where s is the context tree structure). Model selection has been well studied in the past decades, and one popular approach is Bayesian model selection, where candidate structures are evaluated according to their Bayesian marginal likelihood. An alternative approach is based on the Minimum Description Length principle [4], and uses the Normalized Maximum Likelihood distribution [5] or approximations thereof [6] as structure score. In the simplest case, model selection is based on the idea that all observations in the data set follow the same distribution, even though the parametric form of the distribution is not known. Another problem arises if we drop that iid assumption and assume each data point to be generated from one out C possible distributions. General examples for this setting are mixture models and hidden Markov models, more specialized applications are promoter models in computational biology [7, 8]. Parameter learning in a setting with latent variables is well studied, and one popular solution is the EM algorithm [9], which can be perceived as a soft clustering algorithm. Combining the latent variable and model selection problem in the same framework is comparatively unexplored, though. One example are mixture models where each component c also comprises a structure parameter sc, which determines the precise parameterization of the distribution of the component. If all candidate models within a component have the same dimensionality, extending the EM approach for learning latent variable models can by used for learning both the structure and the parameters of each component as shown for a mixture of tree models [10]. However, further extending this approach to model classes where candidate models have different dimensionality [11] is not as straightforward, since it can be shown that asymptotically the largest candidate structures will be selected. Here, we follow an alternative approach by extending the model selection for full observations to the presence of latent variables instead of extending the latent variable learning of fixed-structure models to variable structures.